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Identifying & Mitigating Hazards Tce ai 
= ISO 26262: Hazard and Risk Analysis (HARA) 


e Identify and mitigate risks in accordance with ASIL requirements 








4 1 | 
ie ISO D1 4A8: ee == Known unsafe — (Area 2) 
: . eee Gy Known safe scenarios (Area 1) 
Identify and mitigate 
unsafe scenarios 
e Safety of the Intended 
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Unknown unsafe scenarios (Area 3) 


<=" Unknown safe scenarios (Area 4) 


Unsafe Safe 





Function (SOTIF) — GY 
e Reduce “unknown : emi 
unsafe” area AS fF —_Unknown ace 
— Restrict ODD if needed CO , MINIMIZE ene 
- DY, (3 UNKNOWN 
e Deploy at acceptable ae ly 
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residual risk 


‘ P Carnegie 
Six Sigma Isn't Enough for Safety ea 
= Key Performance Indicators (KPIs) help with quality 
e Are all functions working? 
e Is the functionality improving? 
e Is the fault rate decreasing? 





= Good KPIs are just a start 
e Six Sigma Quality: 99.99966% (five nines) 
— Better, but not enough for life critical functions 
e Fatal Crash Avoidance: 99.9999999996% (eleven nines) 


— Safety is 1 million times more demanding! = 8.34 sigma 
» (example: 1000 opportunities/mile, 250M miles/fatal crash, 1.50 shift) 
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It's All About The Edge Cases tent 


is Gane in training data can SS meses 
lead to perception failure ~ 
e Safety needs to know: . a 
“Is that a person?” 
e Machine learning provides: 
“Is that thing like the people 
already in my training data?” 
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http://bit.ly/2In4rzj 


animal 8.908 


m Edge Case are surprises 
e You wont see these in training or testing 
=> Edge cases are the stuff you didnt think of! 


https://www.clarifai.com/demo 
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Why Edge Cases Matter Nilo 
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= Where will you be after 1 Billion miles of drive-fix-drive? 


= Assume 1 Million miles between unsafe “surprises” 
e Example #1: 
100 “surprises” @ 100M miles / surprise 
— All surprises seen about 10 times during testing ™ . 
— With luck, all bugs are fixed 





e Example #2: 
100,000 “surprises” @ 100B miles / surprise 
— Only 1% of surprises seen during 1B mile testing 
— Bug fixes give no real improvement (1.01M miles / surprise) 


https://goo.gl/3dzguf 
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Real World: Heavy Tail Distribution ee 


Common Things Edge Cases 


Seen In Testing . Not Seen In Testing 


\Random Independent Arrival Rate (exponential) 


ar Law Arrival Rate (80/20 rule 
. a Tail Lorie Many Different, 


Infrequent Scenarios 
Total Area is the same! 
TOTAL TESTING TIME ————————> 
Humans are good at heavy tail © 2021 Philip Koopman 28 
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PROBABILITY OF SURPRISE 
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From Driver Assist to Automation Ce ie 


= Driver Assistance (Advanced Driver Assistance 
System/ADAS) 
AV GETS 
e Effective driver monitoring THE BLAME 
e Safety credit if low false positives 
- Every activation can be a life saved 
—- Non-activation was driver's fault anyway 


= Automated Vehicle (AV) 


PN DYANS SG] SS) 
e Scenario completeness & coverage SAFETY CREDIT 


e Sensor fusion, perception, prediction 
e Blamed for false negatives in heavy tail 
— Every mistake can be a life lost 
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(Burger King owns this trademark. 
They have not endorsed this slide) 


esla Autopilot Mistakes 
Burger King for Stop Signs, 
and They Transform it into an 
dvertisement! 





Q Florance Gold f§June 25,2020 ® autopilot Burger King Tesla Autopilot © 2021 Philip Koopman 31 
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Human Intuition Isnt Enough Uma 
a Some (perhaps most?) surprises are not obvious to humans 


e Characteristics human test designers think shouldnt matter 


e Rare events humans Know are important but are under- 
represented 
— High visibility clothing 


How good is your ADS at 
knowing it doesnt know? 
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Changing Relevance 
of Perception Defects 





“> Functional safety > SOTIF & system safety 
“* Heavy tail/edge cases determine safety 


“* Need to do something safe for unknown unknowns 
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